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For  unmanned  aircraft  to  share  airspace  with  manned  aircraft,  extensive  testing  is  first 
required  to  ensure  that  such  vehicles  can  fly  safely  with  manned  traffic.  Safe  operation 
includes  not  only  avoiding  collisions  with  other  traffic  but  also  complying  with  the  Federal 
Aviation  Regulations  to  remain  “well  clear”  of  other  traffic.  One  method  for  investigating  the 
safety  of  unmanned  aircraft  operations  is  fast-time  Monte  Carlo  simulation  of  encounters 
between  unmanned  and  manned  aircraft.  As  part  of  that  simulation,  one  must  model  how  the 
pilots  of  unmanned  aircraft  react  to  the  encounters.  To  that  end,  a  stochastic  model  of  realistic 
responses  of  unmanned  aircraft  pilots  is  being  built.  A  preliminary  model  was  formulated 
based  on  a  review  of  existing  literature  on  pilot  decision-making,  and  Human-in-the-Loop 
experiments  are  being  used  to  improve  the  model’s  representation  of  unmanned  pilot 
responses  and  parameterize  its  stochastics  elements.  This  paper  summarizes  the  first  of  those 
experiments,  conducted  in  July  2015,  and  highlights  key  results  that  inform  the  pilot  model. 


I.  Introduction 

In  2012,  Congress  mandated  that  the  Federal  Aviation  Administration  establish  rules  for  the  integration  of  unmanned 
aircraft  systems  (UAS)  into  the  National  Airspace  (NAS).  Doing  so  requires  the  development  of  policies,  procedures 
and  equipment  to  ensure  that  operations  within  the  NAS  can  be  conducted  safely.  Existing  aviation  regulations  require 
that  aviators  “see  and  avoid”  nearby  traffic.  Without  a  pilot  in  the  cockpit,  unmanned  aircraft  rely  on  electronic 
surveillance  systems  to  locate  other  air  traffic  and  relay  this  information  to  the  ground  control  station  where  the  pilot 
observes  and  acts  on  it.  The  Air  Force’s  Common  Airborne  Sense  and  Avoid4  (C-ABSAA)  program  is  developing 
and  testing  Detect  and  Avoid  (DAA)  technologies  that  will  help  the  remote  pilot  fly  the  aircraft  safely.  Furthermore, 
the  C-ABSAA  program  is  supporting  the  work  of  the  RTCA  Special  Committee  228  (SC-228)  to  identify  minimum 
performance  requirements  of  a  DAA  system.  These  initial  minimum  performance  standards  rely  on  the  remote  pilot 
to  determine  and  execute  an  escape  maneuver,  rather  than  an  automatically  determined  and  executed  maneuver 
response. 

A  standard  tool  for  testing  the  safety  and  effectiveness  of  these  technologies  and  requirements  is  modeling  and 
simulation.  Fast-time  Monte  Carlo  simulation  facilitates  cost-efficient  testing  of  numerous  equipment  configurations, 
pilot  responses,  and  encounter  types.  For  these  fast-time  Monte  Carlo  simulations  to  address  the  effectiveness  of  DAA 
with  a  pilot-in-the-loop,  one  must  model  how  unmanned  pilots  respond  when  encountering  other  air  traffic.  Safe 
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operation  in  the  NAS  includes  not  only  avoiding  a  collision  with  another  aircraft  but  also  maintaining  what  is 
considered  a  safe  distance  from  other  aircraft — “well  clear”  in  regulatory  terms  [1].  Maneuvering  the  unmanned 
aircraft  to  remain  well  clear  gives  the  pilot  much  more  leeway  than  an  urgent  collision  situation,  so  the  model  of  the 
pilot’s  responses  must  address  this  decision-making  process. 

A  review  of  published  literature  [2-7]  found  little  about  the  behavior  of  unmanned  aircraft  pilots  which  is  not 
surprising  given  the  relative  novelty  of  such  aircraft.  There  is,  however,  a  relatively  large  body  of  work  on  the  behavior 
of  pilots  of  manned  aircraft.  Much  of  this  literature  outlines  the  results  of  human-in-the-loop  laboratory  experiments. 
The  individual  studies  report  disparate  results,  but  synthesizing  the  results  of  these  studies  suggests  the  following 
maneuver  preference  trends: 

•  Single-axis  vertical  or  horizontal  maneuvers  (e.g.,  climb)  are  preferred  over  multi-axis  maneuvers  (e.g., 
climb  and  turn  together). 

•  Vertical  maneuvers  are  preferred  over  horizontal  maneuvers. 

•  No  preference  between  right  and  left  turns  regardless  of  right-of-way  rules 5. 

•  Airspeed  maneuvers  are  not  preferred. 

One  finding  of  interest  by  Thomas  and  Wickens  [7]  was  that  the  method  of  controlling  the  ownship  aircraft  had  a 
moderating  influence  on  the  preference  for  single-axis  maneuvers  and  the  preference  for  vertical  maneuvers.  Unlike 
most  other  studies,  their  experiment  had  pilots  controlling  their  aircraft  through  a  “strategic  route  planner”  where 
rather  than  identifying  how  the  aircraft  should  maneuver  they  were  to  edit  the  desired  flight  path  by  adding  or  changing 
waypoints.  Some  large  unmanned  aircraft  such  as  the  RQ-4  Global  Hawk  and  its  variants  are  flown  in  just  this 
manner — the  pilot  in  the  ground  control  station  has  no  traditional  flight  controls,  and  the  aircraft  determines  how  to 
maneuver  based  on  the  pilot’ s  instructions  of  where  to  go — thus  their  observations  may  be  relevant  for  modeling  a 
UAS  pilot. 

Another  resource  that  informed  the  development  of  a  pilot  model  was  flight  regulations,  which  describe  how  pilots 
should  react  in  some  circumstances  to  ensure  a  complementary  maneuver  for  both  aircraft  in  a  conflict.  The  Federal 
Aviation  Regulations,  14  CFR  Section  91.113  [8],  describes  right-of-way  rules  for  aircraft  and  how  they  should 
maneuver  to  avoid  one  another,  with  right-hand  turns  encouraged.  If  the  goal  of  the  UAS  pilot  is  to  avoid  loss  of  well 
clear,  one  must  ask  whether  maneuvers  to  achieve  that  goal  are  covered  by  the  right-of-way  conditions  in  the  FAR. 
The  rules  are  simple,  but  the  studies  above  found  little  preference  for  right-hand  turns  regardless  of  the  general 
applicability  of  these  rules.  In  addition,  the  final  clause  in  the  section,  “unless  well  clear,”  introduces  ambiguity  into 
the  rules  because  “well  clear”  is  subjective  and  could  depend  on  the  context  of  the  (non-critical)  situation. 

Due  to  the  limited  data  on  the  behavior  of  UAS  pilots,  two  studies  were  conducted  to  support  model  development: 
first,  a  comprehensive  survey  eliciting  feedback  from  pilots  on  their  preferences  for  decision  making  regarding 
maneuvering  to  avoid  traffic,  the  goal  of  which  was  to  qualitatively  inform  an  initial  structure  for  the  model;  second, 
a  human-in-the-loop  experiment  which  examined  how  pilots  actually  responded  when  confronted  with  various  traffic 
situations.  The  data  collected  during  the  second  study  is  being  used  to  update  the  model  structure  and  to  characterize 
variability  in  the  decision  parameters. 

One  of  the  main  limitations  of  prior  studies  was  sample  size.  To  maximize  the  number  of  participants  in  the 
experiment,  while  being  cognizant  of  the  limited  availability  of  UAS  pilots,  a  low-fidelity  “desktop”  simulator  was 
selected  that  was  easy  to  transport,  thereby  allowing  the  simulation  to  be  brought  to  the  pilots. 

A  second  limitation  observed  in  the  literature  was  a  tendency  for  homogenous  populations — for  practical  reasons, 
often  groups  of  aviation  students  from  a  single  flight  school.  Previous  studies  [9]  suggest  that  maneuver  preferences 
depend  on  the  type  of  UAS  the  pilot  operates,  perhaps  due  to  the  dynamic  limitations  of  that  particular  aircraft  or 
procedures  reinforced  in  flight  training  for  that  aircraft.  To  capture  responses  from  a  variety  of  UAS  pilots,  a 
population  was  selected  that  includes  individuals  with  experience  flying  a  range  of  medium  to  large  UAS  (such  as 
Predator,  Reaper  and  Global  Hawk). 


5  The  FAA  regulations  encourage  right-hand  turns,  specifically  for  converging  aircraft  “When  aircraft  of  the  same 
category  are  converging  at  approximately  the  same  altitude  (except  head-on,  or  nearly  so),  the  aircraft  to  the  other's 
right  has  the  right-of-way,  for  head-on  approaches  “When  aircraft  are  approaching  each  other  head-on,  or  nearly  so, 
each  pilot  of  each  aircraft  shall  alter  course  to  the  right”  and  for  overtakes  “ach  aircraft  that  is  being  overtaken  has  the 
right-of-way  and  each  pilot  of  an  overtaking  aircraft  shall  alter  course  to  the  right  to  pass  well  clear.”  Per  Federal 
Aviation  Regulations,  14  CFR  Section  91.113. 


2 


For  public  release.  Distrubution  A.  Approved  for  public  release:  unlimited  distribution. 


II.  Operational  Context 

Established  in  2013,  RTCA  SC-228  is  working  to  develop  the  Minimum  Operational  Performance  Standards 
(MOPS)  for  DAA  equipment  to  be  used  with  unmanned  aircraft.  The  pilot  model  must  fit  within  the  context  of  the 
SC-228  DAA  MOPS  to  be  useful  towards  identification  of  system  requirements.  Key  operational  concepts  of  the  DAA 
system  from  the  vantage  of  pilot  modeling  are  summarized  here. 

The  DAA  MOPS  currently  being  drafted  by  SC-228  describe  the  overall  DAA  system  environment  and  clarifies 
the  many  interoperating  components  of  the  system.  The  perspective  of  an  individual  unmanned  aircraft  pilot  is 
somewhat  simpler.  The  pilot  sits  in  a  Ground  Control  Station  (GCS)  at  a  workstation  including  aircraft  controls  and  a 
DAA  display.  The  DAA  display  indicates  air  traffic  in  the  vicinity  of  the  unmanned  aircraft  as  perceived  by 
surveillance  systems  onboard  the  unmanned  aircraft — either  from  cooperative  systems  like  the  Traffic  Alert  and 
Collision  Avoidance  System  (TCAS)  that  use  information  broadcasted  by  other  aircraft  or  from  non-cooperative 
sensor  systems  such  as  primary  radar  that  scan  the  surrounding  airspace  for  other  traffic.  Communications  between 
the  pilot  and  the  unmanned  aircraft  may  be  by  direct  radio  link  or  satellite  link.  The  pilot  is  also  in  voice 
communication  with  Air  Traffic  Control  (ATC)  services  to  coordinate  any  course  changes. 

A.  Well  Clear  Definition 

Fundamentally,  the  purpose  of  the  DAA  system  is  to  allow  an  unmanned  aircraft  to  meet  the  regulatory 
requirements  to  “see  and  avoid”  other  aircraft,  by  remaining  “well  clear”  of  other  aircraft.  UAS  will  “see”  with 
electronic  surveillance.  Once  aircraft  is  “seen,”  is  must  be  determined  what  constitutes  well  clear.  Federal  regulations 
pertaining  to  aviation  provide  no  specific  definition  of  this  term;  it  is  left  to  the  pilot’s  judgement.  Thus  one  of  the  first 
challenges  faced  in  developing  requirements  for  DAA  systems  was  to  establish  a  measurable,  quantitative  definition. 
This  was  accomplished  by  a  panel  of  subject  matter  experts  (as  documented  by  Cook  et  al.  [1])  and  subsequently 
adopted  by  the  SC-228.  That  definition  states  that  aircraft  are  well  clear  if  any  of  the  following  conditions  are  true: 

1)  Modified-tau  is  greater  than  35  seconds. 

2)  The  predicted  horizontal  miss  distance  (HMD)  is  greater  than  4000  feet. 

3)  The  current  vertical  separation  is  greater  than  450  feet. 

In  this  definition,  modified-tau  is  a  metric  of  time-to-go  to  the  closest  point  of  approach  in  the  horizontal  plane 
and  is  defined  as 


(DMQD  R  if  R  >  DMOD,  R  <  0 

Tmod  =  j  0  if  R  <  DMOD  (1) 

V  oo  ifR>DM0D,  R>0 

where  R  is  the  horizontal  range,  R  is  the  range  rate,  and  DMOD=4000  feet. 

If  at  any  time  all  three  conditions  are  not  true,  a  well  clear  violation  (WCV),  also  referred  to  as  a  Loss  of  Well 
Clear  (LoWC),  has  occurred. 

B.  Alerting 

To  prompt  the  pilot  that  some  action  may  be  needed  to  prevent  a  WCV,  SC-228  has  adopted  an  alerting  scheme 
that  indicates  conflicting  air  traffic  with  color-coded  icons  and,  in  some  cases,  an  aural  alert  signal.  Alert  criteria  are 
based  on  projected  penetration  of  an  alert  region,  defined  similarly  to  the  well  clear  violation  region,  within  a  certain 
look-ahead  time.  The  alert  level  increases  as  the  proximity  to  a  WCV  increases.  Projected  ownship  and  intruder  states 

are  found  by  constant  velocity  extrapolation  of  the  current  state.  The  alerting  scheme  used  in  this  study  is  summarized 

in  Table  1.  Note  that  SC-228  has  continued  to  evolve  the  alerting  scheme  in  the  time  since  this  experiment  was 
conducted;  in  particular,  the  proximate  traffic  advisory  has  been  eliminated  and  the  threshold  values  have  been  revised. 

Alerting  also  factors  into  the  concept  of  operations.  Ordinarily  the  pilot  of  the  UAS  is  required  to  clear  any 
maneuvers  off-course  with  ATC;  however,  under  a  Warning  Alert  pilots  are  permitted  to  maneuver  without  prior  ATC 
approval. 


C.  Display  Requirements 


3 


For  public  release.  Distrubution  A.  Approved  for  public  release:  unlimited  distribution. 


A  pilot  in  the  cockpit  relies  on  his  or  her  eyes,  communication  with  ATC,  and  perhaps  supported  by  a  cockpit 
traffic  display  to  see  and  avoid  other  air  traffic.  For  the  pilot  of  an  unmanned  system,  electronic  surveillance  replaces 
eyes  and  the  traffic  display  becomes  the  primary  view  of  nearby  air  traffic.  As  defined  by  SC-228,  traffic  displays  can 
be  grouped  into  three  categories  depending  on  the  amount  and  type  of  information  shown  and  the  level  of  decision 
support  offered.  The  most  basic  display  is  termed  an  informative  display — it  shows  air  traffic  on  a  map  and  provides 
basic  information  about  each  aircraft  like  airspeed,  vertical  rate,  and  call  sign.  It  provides  no  explicit  maneuver 

Table  1.  Alerting  scheme  used  to  warn  study  subjects  of  nearby  air  traffic. 


Alert 

Level 

Alert  Type/Criteria 

Aural 

Alert 

Symbol 

0 

None 

None 

A 

1 

Proximate  Traffic  Advisory 

•  85 -second  look-ahead  time 

•  35-second  modified  tau 

•  1.5-nm  HMD/DMOD,  1200-ft  vertical 
separation 

None 

□ 

2 

Preventive  Alert 

•  75 -second  look-ahead  time 

•  35-second  modified  tau 

•  1-nm  HMD/DMOD,  700-ft  vertical  separation 

Tone 

3 

Corrective  Alert 

•  75 -second  look-ahead  time 

•  35-second  modified  tau 

•  0.75-nm  HMD/DMOD,  450-ft  vertical 
separation 

Tone 

4 

Warning  Alert 

•  25 -second  look-ahead  time 

•  35-second  modified  tau 

Tone 

m 

guidance  to  the  pilot.  A  suggestive  display  adds  to  this  some  indication  of  a  range  of  possifflWfflffeuvers  to  avoid  a 
conflict.  Finally,  a  directive  display  gives  the  pilot  a  specific  maneuver  to  resolve  a  conflict  (e.g.,  a  resolution  advisory 
(RA)  from  TCAS). 

To  determine  the  display  type  needed  for  DAA  systems,  a  series  of  human-in-the-loop  experiments  were  conducted 
for  SC-228  (separate  from  the  experiment  described  here)  examining  how  well  pilots  could  stay  well  clear  while  using 
various  display  concepts  [10,11].  These  experiments  demonstrated  that  pilots  were  best  able  to  maintain  separation  if 
the  display  indicated  safe  maneuver  choices  (the  suggestive  displays)  than  if  shown  only  traffic  information.  In  light 
of  these  findings,  SC-228  adopted  suggestive  guidance  as  a  minimum  requirement  for  a  DAA  system. 

III.  Pilot  Survey 

The  first  step  in  identifying  how  unmanned  aircraft  pilots  would  respond  in  air  traffic  situations  was  to  conduct 
interviews.  A  survey  was  developed  with  a  total  of  25  questions  in  several  formats  (e.g.,  multiple  choice,  agree- 
disagree,  free  response)  to  probe  various  aspects  of  the  maneuver  decision  to  include  what  sources  of  information  they 
rely  upon,  what  their  first  choice  of  maneuver  direction  would  be,  how  far  in  advance  of  the  conflict  they  prefer  to 
maneuver,  and  how  they  interact  with  ATC.  The  survey  began  by  explaining  the  quantitative  well  clear  definition  and 
other  terms.  The  respondents  were  instructed  to  answer  the  questions  as  though  they  were  flying  an  unmanned  aircraft 
in  the  NAS  under  instrument  flight  rules  (IFR),  in  radio  communications  with  ATC,  and  with  a  traffic  display 
highlighting  traffic  within  80  seconds  of  a  well  clear  violation. 

The  survey  was  distributed  to  military  unmanned  aircraft  pilots,  and  a  total  of  23  responses  were  received.  The 
following  summarizes  noteworthy  results: 

•  Most  pilots  typically  assessed  potential  conflicts  by  distance  to  point  of  closest  approach  rather  than  time. 

•  When  contacting  ATC  to  coordinate  a  maneuver,  most  request  a  specific  heading  or  a  specific  altitude  rather 
than  just  a  maneuver  direction. 
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•  Time  to  closest  point  of  approach,  distance  to  closest  point  of  approach,  and  vertical  separation  were 
identified  as  the  most  useful  information  for  deciding  when  to  maneuver. 

•  Distance  to  closest  point  of  approach,  vertical  separation,  and  ATC  clearance  were  identified  as  the  most 
useful  information  for  deciding  how  to  maneuver. 

•  If  encountering  traffic  while  climbing  or  descending,  most  preferred  to  level  off  to  avoid  a  conflict. 

•  If  encountering  traffic  while  in  level  flight,  a  slight  majority  (60%)  preferred  to  avoid  the  conflict  by  turning. 

IV.  Human-in-the-Loop  Experiment 

The  stated  preferences  collected  with  the  Pilot  Model  Survey  were  useful  towards  establishing  a  general 
architecture  of  the  pilot  model.  However,  to  actually  parameterize  the  model,  quantifiable  revealed  preferences  were 
collected  through  a  Human-in-the-Loop  (HITL)  experiment  in  which  pilot  subjects  were  asked  to  fly  a  generic 
unmanned  aircraft  in  a  simulator  and  maneuver  to  avoid  nearby  air  traffic. 


A.  Basic  Assumpions 

To  be  consistent  with  the  SC-228  concept  of  operations  and  minimum  requirements,  a  display  with  suggestive 
guidance  was  used  to  inform  the  UAS  pilots’  decision  making  with  ATC  providing  clearance  of  maneuver  requests. 
Pilots  were  informed  that  all  encounters  would  be  one-on-one,  at  altitudes  below  10,000  feet,  and  that  the  intruder  was 
noncooperative  (i.e.,  flying  without  a  transponder  turned  on  and  not  communicating  with  ATC). 


B.  Simulator 

The  experiment  was  conducted  using  a  portable  aircraft  encounter  simulator  and  pilot  workstation  provided  by  the 
U.S.  Air  Force  Simulation  and  Analysis  Facility  (SIMAF).  The  pilot  interface  is  built  around  SIMAF’s  simulation 
software  of  the  dynamic  response  of  the  ownship  aircraft  to  pilot  command  inputs.  Without  intervention  from  the 
pilot,  the  ownship  follows  a  predetermined  flight  path.  The  intruder  aircraft  and  all  background  traffic  follow 
predetermined  trajectories. 

The  pilot  workstation  comprised  two  monitors,  shown  in  Figure  1,  a  keyboard  and  a  mouse.  The  left-hand  monitor 
showed  a  moving  map  and  the  Primary  Flight  Display  (PFD)  with  the  aircraft’s  attitude,  heading,  altitude,  and  speed. 
The  pilot’s  ATC  communication  controls  and  ability  to  declare  all  clear  (i.e.,  end  the  simulation)  were  also  found  on 
the  PFD.  The  right-hand  monitor,  the  DAA  display,  showed  nearby  traffic  and  suggestive  guidance. 


Figure  1.  Multifunction  (left)  and  Detect  and  Avoid  (right)  display  comprising  the  UAS  pilot’s  workstation 


Several  pilot  aids,  consistent  with  SC-228  requirements  (as  of  March  2015),  were  incorporated  in  the  interface  to 
help  the  participant  negotiate  traffic  in  the  experiment.  First,  a  succession  of  visual  and  aural  alerts  was  used  to  warn 
pilots  of  nearby  aircraft  that  may  cause  a  well  clear  violation.  The  alert  levels  and  the  conditions  that  trigger  them 
were  summarized  earlier  in  Table  1. 

In  addition  to  alerts,  the  Omni  Bands  suggestive  guidance  algorithm,  part  of  the  Java  Architecture  for  DAA 
Modeling  (JADEM)  software  suite  developed  at  NASA  Ames  Research  Center  and  provided  by  NASA  for  this  study 
[12],  was  used  to  display  the  predicted  alert  level  if  the  pilot  were  to  turn,  climb,  or  descend — in  other  words,  to 
suggest  maneuver  alternatives  that  the  pilot  could  employ  to  resolve  the  conflict.  The  same  suggestive  guidance  was 
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used  as  one  of  the  concept  displays  in  the  NASA  Part  Task  5  (PT5)  experiment  as  described  by  Rorie  et  al.  [10]. 
NASA’s  experiment  showed  that  pilots  were  more  successful  at  avoiding  well  clear  violations  when  given  suggestive 
guidance.  This  finding  led  to  the  SC-228  decision  that  suggestive  guidance  be  a  minimum  requirement  for  DAA 
displays  noted  above. 

Table  2.  Omni  Bands  suggestive  guidance  color-coding. 


Alert  Level 

Color 

Heading  Bands 

Altitude  Bands 

0  -  None 

Green 

Green 

1  -  Proximate  Traffic  Advisory 

Green 

Green 

2  -  Preventive  Alert 

Dashed  Yellow 

Green 

3  -  Corrective  Alert 

Yellow 

Yellow 

4  -  Warning  Alert 

Red 

Red 

Omni  Bands  heading  suggestions  took  the  form  of  one  or  more  colored  arcs  (“bands”)  on  the  traffic  display’s 
compass  rose  and  an  altitude  tape  to  the  right  of  the  compass  rose.  The  “Fly/No  Fly”  bands  were  colored-coded  to 
indicate  the  threat  level  severity  given  a  particular  heading,  as  shown  in  Table  2.  For  example,  a  green  band  meant 
that  flying  that  heading  would  not  lead  to  any  alerts  (preventive,  corrective  or  warning).  Unlike  directive  guidance 
from  systems  such  as  TCAS,  the  pilot  was  not  required  to  follow  suggestive  guidance;  it  was  intended  to  help  the  pilot 
with  decision-making.  Take  note  in  the  table  of  a  difference  between  the  heading  bands  and  altitude  bands:  a  predicted 
preventive-level  alert  is  colored  dashed-yellow  in  heading  but  green  in  altitude. 

Suggestive  guidance  for  the  vertical  plane  was  shown  with  a  column  of  colored  bands  on  the  right  side  of  the 
traffic  display.  Each  band  corresponds  to  altitudes  above  and  below  the  ownship’s  current  altitude  in  increments  of 
five  hundred  feet.  See  Figure  2  for  an  example  of  the  Omni  Bands  suggestive  guidance  from  our  DAA  display. 

Maneuvers  were  commanded  by  entering  the  desired  heading  and/or  altitude  in  text  boxes  on  the  PFD  and  clicking 
an  ‘execute  maneuver’  button — similar  to  the  strategic  route  planner  mentioned  previously.  Pilots  could  choose  a 
horizontal  maneuver,  a  vertical  maneuver,  or  both  and  were  permitted  to  make  as  many  successive  maneuvers  as 


Figure  2.  Heading  bands  shown  on  inner  range  ring  (left)  and  altitude  bands  (right). 
In  this  example,  pilot  can  turn  right  >5°  to  be  alert  free,  descend  to  2,000  ft.  to  be 
alert  free,  or  maintain  course  or  turn  left  >60°  for  a  preventive  alert 


desired.  Horizontal  maneuvers  were  executed  at  3°/s  and  vertical  maneuvers  at  500  fpm  with  a  generic  ownship 
dynamic  model,  consistent  with  the  SC-228  assumptions. 

Pilots  were  trained  to  coordinate  with  ATC  prior  to  maneuvering  if  possible  via  simulated  ATC  interaction.  When 
ready  to  maneuver,  the  pilot  clicked  a  button  on  the  PFD,  and  after  a  delay  drawn  randomly  from  a  gamma  distribution 
with  mean  11  seconds  (fit  to  ATC  coordination  times  from  NASA’s  PT5  experiment)  a  message  was  displayed 
indicating  that  the  pilot  may  proceed.  There  was  no  voice  communications  and  all  maneuver  requests  were  approved. 
Flight  controls  were  not  disabled  during  this  delay,  so  the  pilot  was  able  to  command  maneuvers  before  receiving 
authorization. 
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C.  Encounters 

Twenty-two  single  intruder  encounters  were  selected  from  a  larger  set  generated  using  the  MIT  Lincoln  Laboratory 
developed  uncorrelated  encounter  model  [13]  for  evaluation  of  a  preliminary  pilot  model.  The  UAS  was  assumed  to 
be  on  an  IFR  flight  plan  heading  due  north  in  all  encounters.  All  encounters  occurred  in  Class  E  airspace  below  10,000 
feet  with  a  noncooperative  intruder:  i.e.,  one  whose  maneuvers  were  not  coordinated  with  Air  Traffic  Control  or  the 
UAS  pilot.  Eighteen  encounters  included  a  nonmaneuvering  intruder;  the  remaining  four  encounters  included  scripted 
maneuvers  by  the  intruder.  Encounter  types  included  head-on,  converging  (horizontally  and/or  vertically),  and 
ownship  overtaking.  All  encounters  included  one  or  more  alerts;  however,  three  encounters  require  no  pilot  input  to 
remain  well  clear.  Encounters  began  when  the  two  aircraft  were  10  nautical  miles  apart,  which  was  usually  well  ahead 
of  the  earliest  alert.  Depending  on  encounter  geometry,  encounters  lasted  from  two  to  eight  minutes. 

Scripted  background  traffic  was  included  in  the  encounters  to  force  the  pilots  to  identify  the  threatening  traffic. 
However,  because  the  model  is  intended  for  single  intruder  encounters,  the  background  traffic  was  far  enough  away 
from  the  encounter  of  interest  that  it  was  unlikely  to  influence  the  pilot’s  maneuver  decisions. 

D.  Procedures 

The  experiment  was  conducted  one  pilot  at  a  time.  Each  test  commenced  with  a  training  briefing  to  explain  the 
controls,  display,  operational  procedures  (e.g.,  coordinating  maneuvers  with  ATC)  and  the  objective  of  avoiding  well 
clear  violations  while  minimizing  course  deviations.  After  the  training  briefing,  the  pilot  was  guided  through  several 
training  encounters  for  hands-on  familiarization  with  the  displays  and  controls  before  beginning  the  test. 

To  explore  the  relationship  between  alerts  and  a  pilot’s  decision  to  maneuver,  the  test  encounters  for  each  pilot 
were  divided  into  two  equal  blocks.  In  one  block  of  encounters,  pilots  were  to  maneuver  only  after  an  alert.  In  the 
other  block  of  encounters,  pilots  were  free  to  maneuver  at  will:  that  is,  at  their  discretion  and  best  judgment  without 
having  to  wait  for  an  alert.  Pilots  were  unaware  of  whether  they  would  be  maneuvering  at  will  or  waiting  for  an  alert 
until  just  prior  to  the  encounter  block  being  introduced  and  trained.  The  order  of  the  blocks,  the  encounters  contained 
in  each  block,  and  the  order  of  the  encounters  within  each  block  were  varied  from  pilot  to  pilot  to  preclude  any 
potentially  confounding  training  effects.  For  flexibility,  the  runs  were  balanced  for  groups  of  six  pilots.  The  total  test 
time,  including  training,  each  block  of  encounters  with  a  break  in  between,  and  a  brief  post-test  survey,  was 
approximately  2.5  hours. 

E.  Test  Subjects 

A  relatively  heterogeneous  group  of  26  test  subjects  were  recruited  for  the  study.  All  test  subjects  were  current  or 
recent  UAS  pilots  from  the  following  duty  stations: 

•  Eight  from  Edwards  Air  Force  Base 

•  Four  from  Beale  Air  Force  Base 

•  One  from  Wright-Patterson  Air  Force  Base 

•  Seven  from  Springfield  (OH)  ANG 

Mean,  Minimum,  and  Maximum  Flying  Experience  Reported  UAS  Experience 


Ai  rcraft  T ype  UAS  Type 

Figure  3.  Flying  experience  of  UAS  pilots  participating  in  the  study. 

•  Two  from  Grand  Forks  (ND)  ANG 
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•  Three  from  NASA  Armstrong  Flight  Research  Center 

•  One  from  MIT  Lincoln  Laboratory 

Figure  3  gives  a  more  detailed  demographic  breakdown  of  the  pilots. 


V.  Results 

Overall,  the  pilots  were  generally  successful  at  remaining  well  clear:  out  of  a  total  of  572  encounters  tested,  there 
were  no  near  mid-air  collisions  (defined  as  separation  of  less  than  500  feet  horizontally  and  100  feet  vertically),  only 
12  well  clear  violations.  Seven  of  those  well  clear  violations  were  a  result  of  intruder  maneuvering.  Pilot  feedback 
after  the  experiment  indicated  that  the  displays  provided  adequate  information  to  maintain  separation  and  that  their 
training  was  sufficient. 

Figure  5  shows  the  recorded  responses  of  all  pilots  for  the  first  encounter.  It  is  clear  that  there  is  significant 
variability  in  maneuver  timing,  maneuver  direction,  and  maneuver  magnitude. 

Pilots  were  permitted  to  maneuver  as  often  as  they  wanted,  and  the  data  was  separated  into  the  first  maneuver 
(categorized  as  the  initial  maneuver)  and  any  subsequent  maneuvers  (categorized  as  update  maneuvers).  Pilots 
executed  a  total  of  838  maneuvers  throughout  the  test.  There  was  at  least  one  maneuver  in  every  case,  so  there  were 
572  initial  maneuvers  and  266  update  maneuvers. 

Pilots  almost  universally  complied  with  instructions  to  coordinate  their  traffic  avoidance  maneuvers  with  ATC. 


A.  Maneuver  Timing 

The  maneuver  timing  results  indicated  a 
marked  difference  in  the  pilot’ s  decision  of  when 
to  maneuver  between  the  two  conditions,  wait 
for  an  alert  versus  maneuver  at  will.  Given  the 
freedom  to  do  so,  as  shown  in  Figure  4,  the  pilots 
frequently  maneuvered  away  from  the  intruding 
traffic  before  an  alert.  In  62%  of  “maneuver  at 
will”  encounters  the  initial  maneuver  occurred 
before  the  first  aural  alert  (preventive  or 
corrective  alert,  depending  on  the  encounter). 
We  concluded  that,  lacking  any  disincentive, 
pilots  would  generally  prefer  to  maneuver  upon 
identifying  a  potential  conflict  with  another 
aircraft  whether  or  not  an  alert  has  been  issued. 

Among  encounters  with  the  initial  maneuver 
after  the  first  aural  alert,  the  average  time  from 
the  alert  until  the  pilot  contacted  ATC  was  6.2 
seconds.  Examining  the  response  times  by 
encounter  revealed  that  responses  were  typically 
slower  in  encounters  in  which  the  ownship 
and/or  intruder  was  climbing  or  descending.  The 
average  response  time  in  those  encounters  was 
8.6  seconds  while  the  average  response  time  for 
encounters  with  level  ownship  and  intruder  was 
about  half  that  at  4.4  seconds,  a  significant 
difference  with  p<0.01. 

Since  pilots  maneuvered  in  every  encounter, 
we  could  not  determine  the  encounter 
characteristics  motivating  a  decision  not  to 
maneuver. 

B.  Maneuver  Direction 


Frequency  of  Maneuvers  by  Alert  Level 


Alert  Level  at  Initial  Maneuver 

Figure  4.  Alert  level  at  initial  maneuver  over  all 
encounters. 


Position  East  (nm)  Time  (seconds) 

Figure  5.  Ensemble  of  pilot  responses  to  encounter  #1. 
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Maneuver  Preferences  -  All  Maneuvers 


Figure  6  shows  the  overall  breakdown  of 
maneuver  choices  by  maneuver  plane,  revealing  that 
multi-axis  maneuvers  (‘combo’)  were  substantially 
less  common  than  horizontal  or  vertical  single-axis 
maneuvers,  consistent  with  earlier  studies  noted 
above.  Breaking  these  results  down  further,  one  key 
finding  is  a  clear  distinction  between  the  initial 
maneuvers  selected  prior  to  alert  and  the  initial 
maneuvers  that  occurred  after  alert:  the  former  group 
showed  much  greater  preference  for  vertical 
maneuvers.  The  latter  group  instead  showed 
statistically  significant  preference  for  horizontal 
maneuvers  over  vertical  maneuvers  (p<0.01) — see 
Figure  7.  Furthermore,  there  was  a  significantly 
greater  preference  for  multi-axis  maneuvers  when  the 
initial  maneuver  occurred  after  alert.  Figure  8 
compares  maneuver  preferences  for  initial  maneuvers 
with  those  for  update  maneuvers  over  all  encounters. 

Grouped  this  way,  a  marked  preference  for  turns  in  update  maneuvers  is  evident  while  there  is  not  a  significant 
preference  for  initial  maneuvers.  We  expect  this  relates  to  the  dynamic  capabilities  of  the  ownship — when  close  to 
well  clear  violation,  its  low  maximum  vertical  rate  make  horizontal  maneuvers  more  efficacious.  Of  further  note  in 
Figure  8,  multi-axis  maneuvers  were  rarely  used  for  updates. 
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Figure  7.  Maneuver  plane  preferences  for  initial  maneuvers  executed  before  an  alert  (left,  N=186)  and  after 
an  alert  (right,  N=386). 

Among  vertical  maneuvers,  there  was  a  strong  preference  for  climbs  over  descents  (p<0.01).  However,  the 
encounter  geometry  may  play  a  role  in  that  bias  because  the  intruder  is  level  with  or  below  the  ownship  in  all  but  one 
of  the  level-level  encounters  and  in  many  of  the  encounters  the  ownship ’s  altitude  is  3000  feet.  Since  the  pilots  were 
experienced  at  flying  UAS  to  much  higher  altitudes  it  is  possible  that  this  was  their  preference  and  they  were  wary  of 
descending  too  low  for  traffic  avoidance. 

Among  all  of  the  horizontal  maneuvers,  there  was  a  small  but  statistically  significant  preference  (p<0.03)  for  left 
turns  over  right.  This  preference  is  surprising  because  the  right-of-way  rules  generally  favor  right-hand  turns.  Of 
particular  note,  the  pilots  in  this  experiment  strongly  favored  left-hand  turns  in  head-on  encounters,  turning  to  the  right 
per  the  right-of-way  rules  in  only  29.6%  of  those  encounters.  We  believe  this  has  to  do  with  the  particular  encounter 
geometries  faced  by  the  pilots.  Encounters  were  categorized  as  head-on  when  the  bearing  of  the  intruder  was  within 
10°  of  the  ownship’ s  heading  and  the  bearing  of  the  ownship  was  within  10°  of  the  intruder’s  heading.  All  of  the  so- 
designated  encounters  were  slightly  skewed  and  not  directly  head-on  (i.e.,  opposite  headings  on  collision  course). 
Consequently,  pilots  may  have  mentally  projected  the  intruder’s  trajectory  forward  and  found  that  turning  to  the  left 
was  safer  than  to  the  right.  It  is  not  certain  whether  the  pilots  considered  the  right-of-way  rules  in  reaching  a  decision. 
(The  pilots  were  not  explicitly  instructed  to  follow  right-of-way  rules.) 


Initial  Maneuver  Preferences  -  Before  Alert  Initial  Maneuver  Preferences  -  After  Alert 

_ T _ T _ T _  1  _ T _ T _ 


0.8 


Horizontal  Vertical  Combo  Horizontal  Vertical  Combo 


Horizontal  Vertical  Combo 


Figure  6.  Overall  maneuver  direction  choices 
(N=838). 
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Initial  Maneuver  Preferences 


Update  Maneuver  Preferences 


Figure  8.  Maneuvering  plane  preferences  for  initial  maneuvers  (left,  N=572)  and  update  maneuvers  (right, 
N=266). 

We  also  observed  that  the  direction  of  maneuver  per  axis  correlates  well  with  the  Omni  Bands  suggestions.  When 
pilots  did  turn,  about  75%  of  the  time  it  was  in  the  direction  of  the  smaller  maneuver  suggested  by  Omni  Bands  (e.g., 
left  if  green  is  displayed  for  headings  greater  than  30°  to  the  left  and  45°  to  the  right). 


C.  Maneuver  Magnitude 

Figure  5  displayed  the  wide  range  of  maneuver  magnitudes  selected  by  the  pilots  to  just  one  encounter.  Examining 
maneuver  magnitudes  across  all  encounters  revealed  no  discernable  pattern.  However,  plotting  the  maneuver 
magnitude  choices  relative  to  the  smallest  maneuver  suggested  by  Omni  Bands  in  the  direction  of  the  maneuver  reveals 
a  gamma-like  distribution  (see  Figure  9).  Pilots  usually  selected  maneuvers  10-30°  or  500-1000  ft  larger  than  the 
Omni  Bands  suggestion,  but  in  the  distribution  tails  we  see  evidence  of  the  abundantly  cautious  and  the  overly 
aggressive  pilots  alluded  to  in  Section  1.1.  In  fact,  some  of  the  maneuvers  selected  were  smaller  than  the  Omni  Bands 
suggestion.  There  are  several  potential  explanations  for  this.  First,  the  pilot  may  have  simply  misread  the  guidance. 
This  is  mostly  likely  for  horizontal  maneuvers  as  the  suggestions  are  given  in  1°  increments  but  the  compass  is 
graduated  in  45°  increments  (see  Figure  2).  Second,  the  guidance  may  have  changed  between  when  the  pilot  selected 
his  maneuver  and  when  it  was  executed.  Third,  the  pilot  may  not  have  been  actively  using  the  suggestive  guidance. 
This  is  more  likely  for  vertical  maneuvers  as  the  guidance  is  shown  on  the  side  of  the  DAA  display;  in  fact,  several 
pilots  noted  that  it  was  hard  to  see  the  altitude  bands  when  they  were  focused  on  the  ownship  and  intruder  tracks 
shown  towards  the  center  of  the  display.  Finally,  the  horizontal  extent  of  the  alert  region  is  larger  than  the  extent  of 
the  WCV  region  (0.75  nm  vs.  0.66  nm),  so  a  pilot  could  choose  to  maneuver  to  a  heading  that  will  result  in  an  alert 
but  not  a  WCV. 


Figure  9.  Maneuver  magnitudes  selected  relative  to  the  magnitude  of  the  smallest  OmniBands  suggestion. 
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D.  Post-test  Survey 

At  the  end  of  all  trials,  each  pilot  participant  completed  a  brief  post-test  survey.  The  primary  purpose  of  the  survey 
was  to  ensure  the  validity  of  the  data  collected.  In  fact,  the  pilots  reported  almost  unanimous  strong  agreement  to 
questions  about  whether  they  had  the  training  and  the  information  to  be  able  to  maintain  separation  and  minimize 
course  deviations.  However,  a  question  about  whether  their  strategy  was  affected  by  having  to  wait  for  an  alert  before 
maneuvering  elicited  varied  responses.  This  presents  a  challenge  determining  the  appropriate  triggering  mechanism 
for  the  pilot  model,  and  it  suggests  a  conflict  that  must  be  addressed  in  a  final  concept  of  operations.  Many  pilots  also 
responded  that  they  felt  the  suggestive  guidance  had  not  strongly  influenced  their  decision-making,  while  above  we 
noted  a  compelling  correlation  between  the  actual  responses  and  the  suggestive  guidance. 

VI.  Future  Work 

This  dataset  will  form  the  foundation  of  an  empirically  based  stochastic  operator  model.  Immediate  modeling  goals 
are  to  update  an  existing  heuristic  model  with  preferences  observed  in  the  data;  in  the  longer  run,  machine  learning 
techniques  will  be  employed  to  more  fully  explore  the  dataset  and  identify  correlations  between  encounter  features 
and  pilot  decisions. 

Further  HITL  experimentation  is  planned  to  enhance  fidelity  and  focus  on  particular  situations  of  interest. 
Additional  situational  features  that  may  influence  pilot  behavior  will  be  considered  including: 

•  Imperfect  surveillance. 

•  More  extensively  maneuvering  intruders. 

•  More  realistic  procedures  (e.g.,  requiring  a  return  to  course). 

•  Differing  ownship  dynamic  capabilities  representing  specific  UAS  (e.g.,  Predator  vs.  Global  Hawk). 

VII.  Conclusion 

This  HITL  experiment  provided  a  rich  set  of  data  from  which  to  build  a  model  of  unmanned  aircraft  pilots’ 
responses  to  nearby  air  traffic.  The  experiment  has  identified  several  characteristics  of  UAS  pilot  responses  that  might 
be  considered  for  that  model.  First,  pilots  seeing  a  conflict  coming  would  prefer  to  maneuver  even  earlier  than  the 
alert  timeline  used  in  the  experiment.  Second,  pilots  responding  before  an  alert  when  guidance  was  displayed  prefer 
to  maneuver  in  the  vertical  plane  while  those  closer  to  the  conflict  tend  to  pick  either  a  horizontal  maneuver  or  a 
horizontal/vertical  combination  maneuver.  Third,  response  time  slows  when  the  ownship  and/or  intruder  is  climbing 
or  descending.  The  data  collected  provides  evidence  of  pilot  response  under  many  encounter  conditions;  however, 
further  experimentation  is  necessary  for  more  definitive  answers  to  some  questions.  In  particular,  more  work  is 
required  to  probe  pilot  responses  when  they  are  very  close  to  a  well  clear  violation. 

It  is  important  to  note  that  few  pilots  of  unmanned  aircraft  have  experience  with  UAS  DAA  systems  or  contending 
with  other  air  traffic  because  UAS  operations  in  the  NAS  are  currently  authorized  by  special  exception  only.  The 
responses  recorded  in  this  experiment  are  thus  responses  to  an  unfamiliar  task.  UAS  pilot  behavior  may  evolve  as 
operation  of  UAS  in  the  NAS  becomes  more  common,  and  DAA  systems  are  fielded  with  new  policies,  procedures, 
and  training. 
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